Accuracy measurement of different marker based motion analysis systems for biomechanical applications: A round robin study

Introduction Multiple camera systems are widely used for 3D-motion analysis. Due to increasing accuracies these camera systems gained interest in biomechanical research areas, where high precision measurements are desirable. In the current study different measurement systems were compared regarding their measurement accuracy. Materials and methods Translational and rotational accuracy measurements as well as the zero offset measurements of seven different measurement systems were performed using two reference devices and two different evaluation algorithms. All measurements were performed in the same room with constant temperature at the same laboratory. Equal positions were measured with the systems according to a standardized protocol. Measurement errors were determined and compared. Results The highest measurement errors were seen for a measurement system using active ultrasonic markers, followed by another active marker measurement system (infrared) having measurement errors up to several hundred micrometers. The highest accuracies were achieved by three stereo camera systems, using passive 2D marker points having errors typically below 20 μm. Conclusions This study can help to better assess the results obtained with different measurement systems. With the focus on the measurement accuracy, only one aspect in the selection of a system was considered. Depending on the requirements of the user, other factors like measurement frequency, the maximum analyzable volume, the marker type or the costs are important factors as well.


Introduction
Marker based motion capture analysis is a common approach to make three dimensional human motions visible. Research areas, in which motion analysis is applied, include the analysis and optimization of training methods in sports as well as the examination of human motion for health reasons [1][2][3]. For these applications, the markers are attached to the skin of human subjects on top of bony landmarks and the joint angles during motion can be calculated with different models. Because the skin with the attached markers is moving relative to the anatomical landmarks during motion and a model to determine the joint centers always contain errors, resulting measurement errors of a few millimeters and a few degrees are unavoidable [2,4,5]. Algorithms were developed to limit errors due to soft tissue motion and increase the measurement accuracy [6]. However, in the mentioned research fields, typically the measurement accuracy does not need to be beyond one millimeter or one degree. In biomechanical studies with human specimens, where the kinematics of an isolated joint is analyzed during passive motion, a higher precision of the motion analysis is needed. Small differences of the joint kinematics can help to optimize implant designs or surgical methods. For biomechanical studies, optical markers can directly be attached to the bone using pins building a rigid unit [7][8][9]. Therefore, measurement artefacts as when markers are attached at the soft tissue can be prevented, leading to a higher accuracy [4]. Another biomechanical research field next to the kinematics of the joint is the measurement of micromotions between an artificial joint and a bone of a human specimen, in order to determine the primary stability of the implant [10,11]. For this purpose, markers are attached to the implant as well as to the bone to measure the micromotions between the components under loading conditions. The translational motions between the implant and the bone are often below 100 μm and rotations far below one degree can occur [12][13][14]. Due to improvements of marker-based measurement systems in the last decades, these systems were also used for these biomechanical research areas, where high accuracies are needed.
When acquiring a new optical measurement system for minor motions, the accuracy in all six degrees of freedom of the system is important. The manufacturers of a measurement system typically state either no accuracy or an accuracy, which mostly refers to an in-house measurement of the company in one axis under optimum conditions. Therefore, the aim of this round robin test was to compare the measurement accuracies of different available motion capture systems, when experienced users carry out the same measuring task. The results can help to detect possible variances of the measurement systems, make studies more comparable and should give scientists an overview of the accuracy of some commonly used marker based measurement systems.

Materials and methods
In total, seven different measurement systems from six biomechanical laboratories located in Germany were tested regarding the measurement accuracy and the precision. For this end, system set up the measurement system as they usually do for laboratory tests. Depending on the measurement system, a calibration was performed before the measurement. After that, the measurements were performed with both reference devices successively.

Reference devices and measurement sequences
Two different reference devices were used, in order to ensure the accuracy of the test results and to cover two different measuring ranges. In addition, the marker motion of the first reference system was measured in relation to another fixed marker and the marker motion of the second reference system was determined in relation to the same marker in the zero position.
Coordinate measurement machine. The coordinate measurement machine (CMM, MS222, Mahr, Göttingen, Germany) is positioned in the corner of the measurement room and has an accuracy of 2.0 μm. The size of the measurement room is 17 m 2 floor space and 2.5 m height. The baseplate of the CMM can move in the x-and z-direction, whereby a touch sensor holder can move in the y-direction. The CMM was used for the accuracy measurement of greater distances (0.1 to 100 mm) and to determine the zero offset of the measurement devices.
Two adapters, having a flat rectangular surface, were mounted to the CMM, used as attachment for the different markers. One adapter was fixed at the baseplate and the other one at the touch sensor holder. To measure the accuracy as well as the zero offset of the x-axis, y-axis and z-axis, six movement routines were executed successively. Every measurement started at the home position. In the home position, the adapter for the y-axis was just above the adapter for the x-axis and z-axis (Fig 2). In order to reduce vibrations, the feet of the CMM are made of elastic rubber.
To determine the zero offset (1) a measurement was performed at the home position, then (2) the adapter was driven 10 mm in one axial direction, (3) a measurement was performed again and afterwards (4) the adapter was driven back to the home position and (5) another measurement was performed. This process was repeated five times for statistical purposes. After that, the same protocol was applied for the other two axes. Thus, for each axis, six measurements were performed at zero position and five measurements were performed at a distance of 10 mm. The distances from the moving adapter were determined in relation to the markers of the motionless adapter and compared to the real motion of the CMM to calculate the measurement errors.
In order to measure the accuracy for different distances, the same approach was used as for the previous measurement, but the three axes were driven successively to different positions from the starting position (Table 2) and this procedure was measured five times for statistical purposes.
After each position, a measurement was performed. Using this method, the measurement errors could be determined from very low motions (0.1 mm) to larger distances (100 mm).
Manual reference device. The manual reference device is a six degree of freedom adjustment unit, consisting of three linear bearing stages M-443 including the matching micrometer screws SM-50 (Newport Corporation, Irvine, California, USA) having a measurement sensitivity of 1 μm each and three different rotating bearings. The rotating bearings were Newport M-GON40-L for rotations around the x-axis (sensitivity of 5 arcsec), Newport M-GON40-U for rotations around the y-axis (sensitivity of 8 arcsec) and Newport RS-65 for rotations around the z-axis (sensitivity of 11 arcsec).
The unit can move in one direction or around one axis by drilling a specific micrometer screw (Fig 3).  The manual reference device was fixated at a stable metal frame, which was positioned next to the CMM in the same measurement room. The different markers were fixated on the front plate of the reference device. To calculate the measurement errors, the distances of the markers on the moving reference device were determined relative to the same markers in the zero positions. Therefore, no reference markers on any motionless part were needed.
The six axes were driven successively to different positions by the same user ( Table 3).
The given values of Table 3 correspond to full rotations of the micrometer screws in order to reduce possible user depending variabilities. Measurements were taken at each of the indicated positions. The measurements were repeated five times for statistical purposes.

Statistics
The measured values with the measurement devices were compared with the values of the CMM and with the values of the manual adjustment unit to determine the measurement errors with both reference devices. The mean and standard deviation of the five measurements were calculated for every position. All statistical analyses were carried out using SPSS 22 (IBM, Amonk, NY, USA).

Results
Some of the measurement systems could determine motions in one direction (Pontos, Core and Aramis) by defining a coordinate system in the associated software. The Optotrak and OptiTrack systems used a rigid reference marker body to define the coordinate system. On the other hand, the two measurement devices Q-400 and CMS measured a resulting vector motion from x-, y-and z-translation due to the usage of a global coordinate system. In order to compare the measurement values of the different measurement devices all together, the resulting measured motions of all axes were used and compared to the true values. Because of high differences regarding the measurement errors a logarithmic scale was used for all figures without the presentation of the standard error.

Zero offset measurement results using the coordinate measurement machine
The measurement errors of the zero offset measurements are shown in Fig 4. Zero_X represents five measurements when driving from 10 mm in x-direction to the home position. P_X represents five measurements when driving from the home position 10 mm in x-direction. Zero_Y, P_Y, Zero_Z and P_Z represent similar measurements like Zero_X and P_X but for movements in y-and z-direction.
The total values of the zero offset measurement errors and the associated standard deviation of the zero offset measurements are shown in Table 4.

Accuracy measurement results using the coordinate measurement machine
The measurement errors of the accuracy measurements in x-, y-and z-direction using the CMM are shown in Fig 5. The target point P6 of the measurement in z-direction for the Opti-Track measurement system was deleted because one marker of the rigid body was covered and could not be detected.
The total values of measurement errors of the accuracy measurements in x-, y-and z-direction using the CMM are shown in Table 5.

Accuracy measurement using the manual reference device
The results of the measurements using the manual adjustment unit are separated into translational and rotational results. All adjustments of the manual device for each group were performed by the same experienced user to account for interpersonal variability.
Translational accuracy measurement using the manual reference device. The measurement errors of the translational accuracy measurements in x-, y-and z-direction using the manual measurement device are shown in Fig 6. The total values of the measurement errors and the associated standard deviation of the translational accuracy measurements in x-, y-and z-direction using the manual measurement device are shown in Table 6.
Rotational accuracy measurement using the manual measurement device. The measurement errors of the rotational accuracy measurements around the x-, y-and z-axes using the manual measurement device are shown in Fig 7. The total values of the measurement errors and the associated standard deviation of the rotational accuracy measurements around the x-, y-and z-axes using the manual measurement device are shown in Table 7.

Discussion
In the current study seven different optical measurement systems were compared regarding their accuracies and zero offset in a round robin test. The measurement systems can be divided into two groups: the ones using active markers, and those using passive reflective markers (Q-400, Pontos, OptiTrack, Core and Aramis). The systems with active markers use either ultrasonic (CMS) or optical infrared (Optotrak) markers. The Optotrak system can be equipped with up to eight position sensors. The measurement systems using passive markers can also be divided in two subgroups. Three systems are stereo camera systems from the same manufacturer and use round passive marker dots (Pontos, Core and Aramis). Two systems can be equipped with multiple cameras, whereby one system uses rigid bodies with spherical markers (OptiTrack) and one uses a speckle pattern for motion detection (Q-400).

Coordinate measurement machine
The results of the zero offset measurements revealed the highest measurement errors for the CMS system for all three directions (mostly above 150 μm). The Optotrak and the OptiTrack system measurement errors were above 100 μm when driving to the position P_Z in z-direction. All other systems showed measurement errors below 20 μm for all directions. Similar results could be seen for the step-wise translational measurements in the three directions. Except of one position in y-direction (P1) and one position in z-direction (P8), the CMS system had measurement errors of more than 100 μm. The Optotrak system clearly shows an increase in measurement error when increasing the distance, ending with the highest measurement error for P8 and P9 in y-direction (>400 μm) and z-direction (>800 μm) of all measurement systems. The OptiTrack system shows the highest measurement errors of the passive marker systems, except for P1 in x-direction and P7, P8 and P9 in y-direction. In addition, the OptiTrack system shows the highest measurement error of all systems in the z-direction for P7 (>700 μm). The Q-400 system showed, especially for small movements (P1 to P5), good measurement accuracies with errors below 20 μm in all directions. Longer distances resulted in higher measurement errors ending in errors between 59 and 87 μm for P9 in the three directions. Both systems using multiple cameras (OptiTrack and Q-400) showed a trend  of an increasing measurement error, when increasing the distance. The measurement errors of the Pontos, the Core and the Aramis systems were below 20 μm for all measurement positions in all three directions. The three stereo camera systems and the CMS system did not show an increasing measurement error with an increasing distance. These results show that the stereo camera systems had the highest measurement accuracies for the zero offset and translational motions up to 100 mm. For longer distances however, no statements can be made.

Manual reference device
The translational measurement errors of the CMS system were the highest when using the manual reference device, having errors of more than 70 μm for all positions in x-direction and errors of more than 200 μm for every position in the other two directions. The second highest measurement errors were detected by the Optotrak system with measurement errors between 10 μm and 82 μm for all positions of the three directions, whereby an increased error for an increased distance can be seen as for the measurements using the CMM. All measurement systems using the passive markers showed measurement errors below 20 μm for the translation in y-direction. For the translation in x-direction, all passive marker systems had measurement errors below 20 μm, except for the OptiTrack system, which showed measurement errors between 20 μm and 30 μm. For the translation in z-direction, all passive marker systems had higher measurement errors than for the other directions, but still below 40 μm. Thus, for very small translational motions, the three stereo camera systems and the Q-400 System showed the lowest measurement errors. The rotational measurement errors were also the highest for the CMS system for rotations around the y-axis and z-axis. The highest rotational measurement errors for motions around the x-axis were seen for the OptiTrack system followed by the CMS system. The rotational measurement errors for the three stereo camera systems were lowest, not exceeding 5 arcmin. For biomechanical applications, where typically translations and rotations occur, the translational and rotational measurement errors could sum up, which may lead in even higher total measurement errors.

Comparison of the two reference systems
For the CMM as reference system, the distance of the moving markers was compared relative to a fixed marker. For the manual reference system, the distance of the moving markers was determined relative to the same markers in the zero position. Thus, two different evaluation algorithms were used. The CMM is a user independent and reproducible device, which makes it a suitable tool to determine measurement errors of different measurement systems. However, only translational motions can be applied because the CMM only possesses linear motors. The largest movements can be performed in x-direction followed by the z-direction followed by the y-direction. The maximum translation of all axes was determined using the maximum motion of the y-direction. The manual reference device enables the comparison of translational and rotational motions between different measurement systems. However, due to the necessity of a user to adjust the different rotational and translational motions, this system is more prone to individual errors. In addition, only small translational motions can be applied due to the limit of the micrometer screws. For users of camera based motion capture systems who don't have any high precision reference systems like the CMM, the manual reference device can be a rational and relatively inexpensive option. The translation in x-, y-and z-direction of P4, using the CMM, and P3, using the manual reference system, were both 5 mm in total. Therefore, the measurement errors P4 of the CMM and P3 of the manual device can be compared. For the Optotrak, Pontos, Core and Aramis systems slightly higher errors were detected in the measurements using the manual system in comparison with the CMM. For the OptiTrack system it is the other way round. For the CMS and Q-400 system, the CMM at P4 provoke slightly higher measurement error compared to P3 of the manual device in x-direction, contrarily to the y-and z-directions. However, marker attachment, the distance of the measurement systems to the markers or other environmental factors may have a higher impact on the measurement errors than the reference system or the evaluation algorithm.

Comparison of the different measurement systems
In general, the two measurement systems using active markers showed the highest measurement errors for both reference devices. Comparing these two measurement systems, the Optotrak revealed better measurement accuracies than the CMS system. In the case of the Optotrak, the measurement did not take place in the optimal measurement volume calibrated by the manufacturer. The measurement in the calibrated measurement volume was not possible due to the size of the air-conditioned measurement room. The highest measurement errors for the passive markers systems were seen for the OptiTrack system. The Q-400 system showed good results for all motions, especially for small movements up to 5 mm. The OptiTrack system used seven and the Q-400 system used three cameras. The accuracy could decrease or increase by reducing or adding cameras, depending on the requirements [4]. The best results for the translational and rotational motions were seen for the three stereo camera systems (Pontos, Core and Aramis), which were all from the same manufacturer and use 2D reference point markers. A comparison of different motion capture systems was published by Richards in 1999 using a self-constructed motorized device to change the passive maker positions [15]. Topley and Richards recently repeated the measurements to compare the measurement accuracy of modern optoelectronic motion capture systems with the systems 20 years earlier [16]. They showed that the development of modern motion capture systems led to an advanced measurement accuracy. Compared to the stereo camera measurement systems within this study, the measurements by Topley and Richards revealed much higher measurement errors. However, they used measurement systems and spherical passive markers which are typically used for human motion capture and the distances between the markers were much greater than in the current test. Topley and Richards used a digitizer to determine the positions of the passive markers having an absolute accuracy of 0.036 mm. For the small measurement errors of the measurement systems in the current study, which are partly below 20 μm or even below 10 μm, an even more precise reference system like the manual adjustment unit and the CMM were needed. The measurement systems in the study by Topley and Richards were typically used for human motion capture analysis, where marker positioning and soft tissue motion provoke higher measurement errors than the systems analyzed [16]. One important parameter next to the accuracy is the maximum analyzable volume. A larger analyzable volume leads to a lower resolution and therefore, to a lower accuracy. As shown in Table 1, the active marker systems and the two measurement systems consisting of multiple cameras cover a larger volume than the stereo camera systems. The volumes of the stereo camera systems can be adjusted by using a bigger calibration plate, but only to a limited extend. Hence, the stereo camera systems are not recommended for human motion capture analysis, where large joint motions should be analyzed. Also a speckle pattern, as used by the Q-400 system, seems to be unsuitable for human motion capture analysis, due to the low depth measurement. However, in the current study, the maximum distance for determining the measurement accuracy was in an area of 100 mm, which is much smaller than the distances during human motion analyses. The main goal was to determine the accuracy for biomechanical measurement, like cadaver joint motions and implant fixation, which need a high accuracy. For analysis of cadaver joint motions, all measurement systems of the current study seem to be useful, but this depends on the extent of the motion. For bigger joint motions the systems, which cover a larger volume are more useful. Typically, the rigid body markers systems offer the user the possibility to select bony landmarks and define an anatomic coordinate system, which is a useful tool for measurement of joint kinematics. However, for biomechanical high precision measurements like micro-motions between implant and bone or measurements of the fracture gaps of bones supplied with osteosynthesis plates under loading conditions, where small volumes are sufficient, the passive stereo camera systems of the current study are of mature interest. Depending on the biomechanical application, the maximum frequency of the measurement system is an important factor as well.

Limitations
The study design has some limitations, which need to be addressed. First, general statements about the measurement accuracy cannot be made, as each measurement system is represented only once. Although every laboratory in the round robin test is a frequent user of their measurement system, results could differ when the calibration, measurement and data analysis with the same measurement system is performed by another user. In the current study the data analysis (moving marker to motionless marker for the CMM and moving marker to the same marker at zero position for the manual reference device) were identical between the groups. Another limitation is the relatively small size of the measurement room (17 m 2 floor space and 2.5 m height). For the Optotrak system which covers a big volume the optimum distance between marker and measurement system was not achieved. Therefore, the measurement accuracy of this system is expected to be higher if the minimum distance is given. For the other measuring systems the minimum distance could be guaranteed.
Ultimately, the CMM seems to be a reliable tool for reference measurements of different measurement systems. It has a similar reproducibility and no user could accidentally influence its motions. However, only translational motions can be driven with the CMM.
The manual system seems to be a simple and useful tool to compare the translational and rotational measurement accuracies of different measurement systems. To account for interpersonal difference one user performed the adjustments of the manual device for every group.
Problematic for the determination of the rotatory accuracies around the different axes was the existence of different rotatory units with different accuracies. This means that a comparison of the measurement errors between the three axes of a measuring system is not meaningful. However, a comparison of the different systems regarding the rotational accuracy can be made. The complete tests were run under optimized conditions and the markers were placed on solid bodies. Therefore, the absolute values of measurement errors determined in this study cannot be transferred directly into a biomechanical application, where the markers are typically fixated on soft tissue or bones.

Conclusions
The results of the current study show advantages regarding the measurement accuracy of single motions up to 100 mm for the measurement systems using passive markers and especially for the stereo camera systems. However, depending on the requirements of the user and the application, other factors like the measurement frequency, the maximum analyzable volume, the marker type and the costs for the measurement system are important factors to be considered.
Supporting information S1 Table. CMM data. Measurement data from the seven measurement systems using the setup with the coordinate measurement machine. Values of the zero offset measurement and for the measurement points approached in the x, y and z directions. (XLSX) S2 Table. Manual reference device data. Measurement data from the seven measurement systems using the setup with the manual reference device. Values for the measuring points approached in the x, y and z directions and the rotations performed about these three axes. (XLSX)